Group-Wise Principal Component Analysis for Exploratory Intrusion Detection

Intrusion detection is a relevant layer of cybersecurity to prevent hacking and illegal activities from happening on the assets of corporations. Anomaly-based Intrusion Detection Systems perform an unsupervised analysis on data collected from the network and end systems, in order to identify singular events. While this approach may produce many false alarms, it is also capable of identifying new (zeroday) security threats. In this context, the use of multivariate approaches such as Principal Component Analysis (PCA) provided promising results in the past. PCA can be used in exploratory mode or in learning mode. Here, we propose an exploratory intrusion detection that replaces PCA with Group-wise PCA (GPCA), a recently proposed data analysis technique with additional exploratory characteristics. A main advantage of GPCA over PCA is that the former yields simple models, easy to understand by security professionals not trained in multivariate tools. Besides, the workflow in the intrusion detection with GPCA is more coherent with dominant strategies in intrusion detection. We illustrate the application of GPCA in two case studies.This work was supported in part by the Spanish Government-MINECO (Ministerio de Economía y Competitividad), using the Fondo Europeo de Desarrollo Regional (FEDER), under Projects TIN2014-60346-R and Project TIN2017-83494-R

Present and Future of Network Security Monitoring

This work was funded by the Ministry of Science and Innovation through CDTI through the Ayudas Cervera para Centros Tecnologicos grant of the Spanish Centre for the Development of Industrial Technology (CDTI) through the Project EGIDA under Grant CER-20191012, and in part by the Spanish Ministry of Economy and Competitiveness and European Regional Development Fund (ERDF) funds under Project TIN2017-83494-R.Network Security Monitoring (NSM) is a popular term to refer to the detection of security incidents by monitoring the network events. An NSM system is central for the security of current networks, given the escalation in sophistication of cyberwarfare. In this paper, we review the state-of-the-art in NSM, and derive a new taxonomy of the functionalities and modules in an NSM system. This taxonomy is useful to assess current NSM deployments and tools for both researchers and practitioners. We organize a list of popular tools according to this new taxonomy, and identify challenges in the application of NSM in modern network deployments, like Software Defined Network (SDN) and Internet of Things (IoT).Ministry of Science and Innovation through CDTI through the Ayudas Cervera para Centros Tecnologicos grant of the Spanish Centre for the Development of Industrial Technology (CDTI) through the Project EGIDA CER-20191012Spanish Ministry of Economy and CompetitivenessEuropean Regional Development Fund (ERDF) funds TIN2017-83494-

On Understanding the Existence of a Deep Torrent

Nowadays, a great part of the Internet content is not reachable from search engines. Studying the nature of these contents from a cyber security perspective is of a high interest, as they could be part of many malware distribution processes, child pornography or copyrighted material exchange, botnet command and control messages, etc. Although the research community has put a big effort on this challenge, most of the existing works are focused on contents that are hidden in Web sites. Yet, there exist other relevant services that are used to keep and transmit hidden resources, such as P2P protocols. In the present work, we suggest the concept of Deep Torrent to refer to those torrents available in BitTorrent that cannot be found by means of public Web sites or search engines. We present an implementation of a complete system to crawl the Deep Torrent and evaluate its existence and size. We describe a basic experiment crawling the Deep Torrent for 39 days, in which an initial estimation of its size is 67% of the total number of resources shared in BitTorrent network

Evaluation of Diagnosis Methods in PCA-based Multivariate Statistical Process Control

Multivariate Statistical Process Control (MSPC) based on Principal Component Analysis (PCA) is a well-known methodology in chemometrics that is aimed at testing whether an industrial process is under Normal Operation Conditions (NOC). As a part of the methodology, once an anomalous behaviour is detected, the root causes need to be diagnosed to troubleshoot the problem and/or avoid it in the future. While there have been a number of developments in diagnosis in the past decades, no sound method for comparing existing approaches has been proposed. In this paper, we propose such a procedure and use it to compare several diagnosis methods using randomly simulated data and from realistic data sources. This is a general comparative approach that takes into account factors that have not previously been considered in the literature. The results show that univariate diagnosis is more reliable than its multivariate counterpart

Fusing Information from Tickets and Alerts to Improve the Incident Resolution Process

In the context of network incident monitoring, alerts are useful notifications that provide IT management staff with information about incidents. They are usually triggered in an automatic manner by network equipment and monitoring systems, thus containing only technical information available to the systems that are generating them. On the other hand, ticketing systems play a different role in this context. Tickets represent the business point of view of incidents. They are usually generated by human intervention and contain enriched semantic information about ongoing and past incidents. In this article, our main hypothesis is that incorporating tickets information into the alert correlation process will be beneficial to the incident resolution life-cycle in terms of accuracy, timing, and overall incident’s description. We propose a methodology to validate this hypothesis and suggest a solution to the main challenges that appear. The proposed correlation approach is based on the time alignment of the events (alerts and tickets) that affect common elements in the network. For this we use real alert and ticket datasets obtained from a large telecommunications network. The results have shown that using ticket information enhances the incident resolution process, mainly by reducing and aggregating a higher percentage of alerts compared with standard alert correlation systems that only use alerts as the main source of information. Finally, we also show the applicability and usability of this model by applying it to a case study where we analyze the performance of the management staff

Semi-supervised Multivariate Statistical Network Monitoring for Learning Security Threats

This paper presents a semi-supervised approach for intrusion detection. The method extends the unsupervised Multivariate Statistical Network Monitoring approach based on Principal Component Analysis by introducing a supervised optimization technique to learn the optimum scaling in the input data. It inherits the advantages of the unsupervised strategy, capable of uncovering new threats, with that of supervised strategies, able of learning the pattern of a targeted threat. The supervised learning is based on an extension of the gradient descent method based on Partial Least Squares (PLS). Moreover, we enhance this method by using sparse PLS variants. The practical application of the system is demonstrated on a recently published real case study, showing relevant improvements in detection performance and in the interpretation of the attacks

PCA-based Multivariate Statistical Network Monitoring for Anomaly Detection

The multivariate approach based on Principal Component Analysis (PCA) for anomaly detection received a lot of attention from the networking community one decade ago mainly thanks to the work of Lakhina and co-workers. However, this work was criticized by several authors that claimed a number of limitations of the approach. Neither the original proposal nor the critic publications were completely aware of the established methodology for PCA anomaly detection, which by that time had been developed for more than three decades in the area of industrial monitoring and chemometrics as part of the Multivariate Statistical Process Control (MSPC) theory. In this paper, the main steps of the MSPC approach based on PCA are introduced; related networking literature is reviewed, highlighting some differences with MSPC and drawbacks in their approaches; and specificities and challenges in the application of MSPC to networking are analyzed. All of this is demonstrated through illustrative experimentation that supports our discussion and reasoning

A Model of Data Forwarding in MANETs for Lightweight Detection of Malicious Packet Dropping

This work introduces a model of data forwarding in MANETs which is used for recognizing malicious packet dropping behaviors. First, different legitimate packet discard situations are modeled, such as those generated by collisions, channel errors or mobility related droppings. Second, we propose an anomaly-based IDS system based on an enhanced windowing method to carry out the collection and analysis of selected crosslayer features. Third, a real deployment of the IDS is also considered by suggesting a methodology for the collection of the selected features in a distributed manner. We evaluate our proposal in a simulation framework and the experimental results show a considerable enhancement in detection results when compared with other approaches in the literature. For instance, our scheme shows a 22% improvement in terms of true positives rate and a remarkable 83% improvement in terms of false positives rate when compared to previous well-known statistical solutions. Finally, it is notable the simplicity and lightweightness of the proposal

Taking the pulse of Earth's tropical forests using networks of highly distributed plots

Tropical forests are the most diverse and productive ecosystems on Earth. While better understanding of these forests is critical for our collective future, until quite recently efforts to measure and monitor them have been largely disconnected. Networking is essential to discover the answers to questions that transcend borders and the horizons of funding agencies. Here we show how a global community is responding to the challenges of tropical ecosystem research with diverse teams measuring forests tree-by-tree in thousands of long-term plots. We review the major scientific discoveries of this work and show how this process is changing tropical forest science. Our core approach involves linking long-term grassroots initiatives with standardized protocols and data management to generate robust scaled-up results. By connecting tropical researchers and elevating their status, our Social Research Network model recognises the key role of the data originator in scientific discovery. Conceived in 1999 with RAINFOR (South America), our permanent plot networks have been adapted to Africa (AfriTRON) and Southeast Asia (T-FORCES) and widely emulated worldwide. Now these multiple initiatives are integrated via ForestPlots.net cyber-infrastructure, linking colleagues from 54 countries across 24 plot networks. Collectively these are transforming understanding of tropical forests and their biospheric role. Together we have discovered how, where and why forest carbon and biodiversity are responding to climate change, and how they feedback on it. This long-term pan-tropical collaboration has revealed a large long-term carbon sink and its trends, as well as making clear which drivers are most important, which forest processes are affected, where they are changing, what the lags are, and the likely future responses of tropical forests as the climate continues to change. By leveraging a remarkably old technology, plot networks are sparking a very modern revolution in tropical forest science. In the future, humanity can benefit greatly by nurturing the grassroots communities now collectively capable of generating unique, long-term understanding of Earth's most precious forests. Resumen Los bosques tropicales son los ecosistemas más diversos y productivos del mundo y entender su funcionamiento es crítico para nuestro futuro colectivo. Sin embargo, hasta hace muy poco, los esfuerzos para medirlos y monitorearlos han estado muy desconectados. El trabajo en redes es esencial para descubrir las respuestas a preguntas que trascienden las fronteras y los plazos de las agencias de financiamiento. Aquí mostramos cómo una comunidad global está respondiendo a los desafíos de la investigación en ecosistemas tropicales a través de diversos equipos realizando mediciones árbol por árbol en miles de parcelas permanentes de largo plazo. Revisamos los descubrimientos más importantes de este trabajo y discutimos cómo este proceso está cambiando la ciencia relacionada a los bosques tropicales. El enfoque central de nuestro esfuerzo implica la conexión de iniciativas locales de largo plazo con protocolos estandarizados y manejo de datos para producir resultados que se puedan trasladar a múltiples escalas. Conectando investigadores tropicales, elevando su posición y estatus, nuestro modelo de Red Social de Investigación reconoce el rol fundamental que tienen, para el descubrimiento científico, quienes generan o producen los datos. Concebida en 1999 con RAINFOR (Suramérica), nuestras redes de parcelas permanentes han sido adaptadas en África (AfriTRON) y el sureste asiático (T-FORCES) y ampliamente replicadas en el mundo. Actualmente todas estas iniciativas están integradas a través de la ciber-infraestructura de ForestPlots.net, conectando colegas de 54 países en 24 redes diferentes de parcelas. Colectivamente, estas redes están transformando nuestro conocimiento sobre los bosques tropicales y el rol de éstos en la biósfera. Juntos hemos descubierto cómo, dónde y porqué el carbono y la biodiversidad de los bosques tropicales está respondiendo al cambio climático y cómo se retroalimentan. Esta colaboración pan-tropical de largo plazo ha expuesto un gran sumidero de carbono y sus tendencias, mostrando claramente cuáles son los factores más importantes, qué procesos se ven afectados, dónde ocurren los cambios, los tiempos de reacción y las probables respuestas futuras mientras el clima continúa cambiando. Apalancando lo que realmente es una tecnología antigua, las redes de parcelas están generando una verdadera y moderna revolución en la ciencia tropical. En el futuro, la humanidad puede beneficiarse enormemente si se nutren y cultivan comunidades de investigadores de base, actualmente con la capacidad de generar información única y de largo plazo para entender los que probablemente son los bosques más preciados de la tierra. Resumo Florestas tropicais são os ecossistemas mais diversos e produtivos da Terra. Embora uma boa compreensão destas florestas seja crucial para o nosso futuro coletivo, até muito recentemente os esforços de medições e monitoramento foram amplamente desconexos. É essencial formarmos redes para obtermos respostas que transcendem fronteiras e horizontes de agências financiadoras. Neste estudo nós mostramos como uma comunidade global está respondendo aos desafios da pesquisa de ecossistemas tropicais, com equipes diversas medindo florestas, árvore por árvore, em milhares de parcelas monitoradas à longo prazo. Nós revisamos as maiores descobertas científicas deste trabalho, e mostramos também como este processo está mudando a ciência de florestas tropicais. Nossa abordagem principal envolve unir iniciativas de base a protocolos padronizados e gerenciamento de dados a fim de gerar resultados robustos em escalas ampliadas. Ao conectar pesquisadores tropicais e elevar seus status, nosso modelo de Rede de Pesquisa Social reconhece o papel-chave do produtor dos dados na descoberta científica. Concebida em 1999 com o RAINFOR (América do Sul), nossa rede de parcelas permanentes foi adaptada para África (AfriTRON) e Sudeste asiático (T-FORCES), e tem sido extensamente reproduzida em todo o mundo. Agora estas múltiplas iniciativas estão integradas através de uma infraestrutura cibernética do ForestPlots.net, conectando colegas de 54 países de 24 redes de parcelas. Estas iniciativas estão transformando coletivamente o entendimento das florestas tropicais e seus papéis na biosfera. Juntos nós descobrimos como, onde e por que o carbono e a biodiversidade da floresta estão respondendo às mudanças climáticas, e seus efeitos de retroalimentação. Esta duradoura colaboração pantropical revelou um grande sumidouro de carbono persistente e suas tendências, assim como tem evidenciado quais direcionadores são mais importantes, quais processos florestais são mais afetados, onde eles estão mudando, seus atrasos no tempo de resposta, e as prováveis respostas das florestas tropicais conforme o clima continua a mudar. Dessa forma, aproveitando uma notável tecnologia antiga, redes de parcelas acendem faíscas de uma moderna revolução na ciência das florestas tropicais. No futuro a humanidade pode se beneficiar incentivando estas comunidades basais que agora são coletivamente capazes de gerar conhecimentos únicos e duradouros sobre as florestas mais preciosas da Terra. Résume Les forêts tropicales sont les écosystèmes les plus diversifiés et les plus productifs de la planète. Si une meilleure compréhension de ces forêts est essentielle pour notre avenir collectif, jusqu'à tout récemment, les efforts déployés pour les mesurer et les surveiller ont été largement déconnectés. La mise en réseau est essentielle pour découvrir les réponses à des questions qui dépassent les frontières et les horizons des organismes de financement. Nous montrons ici comment une communauté mondiale relève les défis de la recherche sur les écosystèmes tropicaux avec diverses équipes qui mesurent les forêts arbre après arbre dans de milliers de parcelles permanentes. Nous passons en revue les principales découvertes scientifiques de ces travaux et montrons comment ce processus modifie la science des forêts tropicales. Notre approche principale consiste à relier les initiatives de base à long terme à des protocoles standardisés et une gestion de données afin de générer des résultats solides à grande échelle. En reliant les chercheurs tropicaux et en élevant leur statut, notre modèle de réseau de recherche sociale reconnaît le rôle clé de l'auteur des données dans la découverte scientifique. Conçus en 1999 avec RAINFOR (Amérique du Sud), nos réseaux de parcelles permanentes ont été adaptés à l'Afrique (AfriTRON) et à l'Asie du Sud-Est (T-FORCES) et largement imités dans le monde entier. Ces multiples initiatives sont désormais intégrées via l'infrastructure ForestPlots.net, qui relie des collègues de 54 pays à travers 24 réseaux de parcelles. Ensemble, elles transforment la compréhension des forêts tropicales et de leur rôle biosphérique. Ensemble, nous avons découvert comment, où et pourquoi le carbone forestier et la biodiversité réagissent au changement climatique, et comment ils y réagissent. Cette collaboration pan-tropicale à long terme a révélé un important puits de carbone à long terme et ses tendances, tout en mettant en évidence les facteurs les plus importants, les processus forestiers qui sont affectés, les endroits où ils changent, les décalages et les réactions futures probables des forêts tropicales à mesure que le climat continue de changer. En tirant parti d'une technologie remarquablement ancienne, les réseaux de parcelles déclenchent une révolution très moderne dans la science des forêts tropicales. À l'avenir, l'humanité pourra grandement bénéficier du soutien des communautés de base qui sont maintenant collectivement capables de générer une compréhension unique et à long terme des forêts les plus précieuses de la Terre. Abstrak Hutan tropika adalah di antara ekosistem yang paling produktif dan mempunyai kepelbagaian biodiversiti yang tinggi di seluruh dunia. Walaupun pemahaman mengenai hutan tropika amat penting untuk masa depan kita, usaha-usaha untuk mengkaji dan mengawas hutah-hutan tersebut baru sekarang menjadi lebih diperhubungkan. Perangkaian adalah sangat penting untuk mencari jawapan kepada soalan-soalan yang menjangkaui sempadan dan batasan agensi pendanaan. Di sini kami menunjukkan bagaimana sebuah komuniti global bertindak balas terhadap cabaran penyelidikan ekosistem tropika melalui penglibatan pelbagai kumpulan yang mengukur hutan secara pokok demi pokok dalam beribu-ribu plot jangka panjang. Kami meninjau semula penemuan saintifik utama daripada kerja ini dan menunjukkan bagaimana proses ini sedang mengubah bidang sains hutan tropika. Teras pendekatan kami memberi tumpuan terhadap penghubungan inisiatif akar umbi jangka panjang dengan protokol standar serta pengurusan data untuk mendapatkan hasil skala besar yang kukuh. Dengan menghubungkan penyelidik-penyelidik tropika dan meningkatkan status mereka, model Rangkaian Penyelidikan Sosial kami mengiktiraf kepentingan peranan pengasas data dalam penemuan saintifik. Bermula dengan pengasasan RAINFOR (Amerika Selatan) pada tahun 1999, rangkaian-rangkaian plot kekal kami kemudian disesuaikan untuk Afrika (AfriTRON) dan Asia Tenggara (T-FORCES) dan selanjutnya telah banyak dicontohi di seluruh dunia. Kini, inisiatif-inisiatif tersebut disepadukan melalui infrastruktur siber ForestPlots.net yang menghubungkan rakan sekerja dari 54 negara di 24 buah rangkaian plot. Secara kolektif, rangkaian ini sedang mengubah pemahaman tentang hutan tropika dan peranannya dalam biosfera. Kami telah bekerjasama untuk menemukan bagaimana, di mana dan mengapa karbon serta biodiversiti hutan bertindak balas terhadap perubahan iklim dan juga bagaimana mereka saling bermaklum balas. Kolaborasi pan-tropika jangka panjang ini telah mendedahkan sebuah sinki karbon jangka panjang serta arah alirannya dan juga menjelaskan pemandu-pemandu perubahan yang terpenting, di mana dan bagaimana proses hutan terjejas, masa susul yang ada dan kemungkinan tindakbalas hutan tropika pada perubahan iklim secara berterusan di masa depan. Dengan memanfaatkan pendekatan lama, rangkaian plot sedang menyalakan revolusi yang amat moden dalam sains hutan tropika. Pada masa akan datang, manusia sejagat akan banyak mendapat manfaat jika memupuk komuniti-komuniti akar umbi yang kini berkemampuan secara kolektif menghasilkan pemahaman unik dan jangka panjang mengenai hutan-hutan yang paling berharga di dunia